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ABSTRACT 



A study assessed the cognitive level of academic challenges 
incorporated into courses offered in the College of Agricultural Sciences at 
the Pennsylvania State University. Eleven faculty members from nine 
departments provided copies of all academic challenges used in their courses. 
They were categorized by type: activities, problem sets, written reports, 
presentations, laboratory tests, quizzes, midterms, and finals. A number 
corresponding to one of the six levels of Bloom's taxonomy (knowledge, 
comprehension, application, analysis, synthesis, evaluation) was assigned to 
each individual task, question, problem, or action. The value of the academic 
challenge to students’ course grade was calculated based on the grade 
weighting information provided in each course's syllabus. Findings indicated 
that the mean cognitive distribution for all the courses' academic challenges 
was fairly even across the six cognitive levels with an emphasis on the 
higher cognitive levels. On average, 31.4 percent of the student's grade came 
from work at the lower cognitive levels and 66.6 from work at the higher 
levels. Individual presentations, quizzes, midterm exams, and final exams 
tended to provide lower cognitive challenges for students. Generally, 
students were rewarded more for completing academic challenges written at the 
higher cognitive levels and less for lower cognitive work. (Contains 12 
references) (YLB) 
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COGNITIVE LEVEL OF ACADEMIC CHALLENGES 



PROVIDED TO COLLEGE STUDENTS 
Introduction 

One of the goals established by The National Education Goals Panel was that, "The 
proportion of college graduates who demonstrate an advanced ability to think critically, 
communicate effectively and solve problems will increase substantially" (1991, p. 5). To meet this 
goal, professors must place greater emphasis on developing students’ ability to think at higher 
cognitive levels. Undergraduate instruction should focus not only on providing students the 
content knowledge of their chosen discipline, but also on facilitating the development of students’ 
critical thinking skills . As stated by Newcomb, 

The need to have students graduate with the demonstrated capacity to 
think at the higher levels of Bloom's taxonomy is more urgent than ever. 

The nature of the world we live in demands it. Given the pace of 
technological change and the unabated explosion of knowledge, it is 
fruitless to try to focus on teaching facts, for this is guaranteed to be a 
losing proposition. (1995, p.4) 

How students leam (i.e., how they are challenged to learn) is a key component of this 
process. If undergraduate students are to develop their ability to think at higher levels of 
cognition, they must be challenged to do so by both professors’ in-class instructional techniques, 
and by the academic challenges (i.e. homework) provided throughout the course. Studies have 
examined faculty teaching characteristics and instructional methods with respect to critical 
thinking skills and opportunities provided by professors for students to engage in higher order 
thinking (Bowman, 1995; Newcomb & Trefz, 1987; Pickford& Newcomb, 1989; Whittington, 
1995; Whittington & Newcomb, 1993). 

Additional studies have shown that effective use of academic challenges can increase 
student achievement (Ziegler, 1986) and can contribute to challenging both students’ progression 
through the thought processes and their development of thinking skills (Meyers, 1986; Terenzini, 
Springer, Pascarella & Nora, 1995). However, there are few studies which have examined the 
cognitive level of academic challenges or which provide a system for assessing academic 
challenges (e.g. Newcomb, 1987; Pickford, 1989; Ratcliff, Jones, Guthrie & Oehler, 1991). 
Because academic challenges represent additional opportunities for learning and the development 
of thinking skills, it is important that they are included in the search for ways to improve 
undergraduate education. There is a need for “systematic assessment and analysis of coursework 
patterns... [which] can be used to launch new, better-informed bases for student advisement, 
teaching, and learning” (Ratcliff et al., 1991). 
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Theoretical Frame 



One of the most frequently cited and applied systems for categorizing cognitive processes 
has been the classification system proposed by Bloom et al. in the Taxonomy of Educational 
Objectives, Handbook I: Cognitive Domain (1956). This taxonomy classifies cognition and 
associated behaviors into six hierarchical levels based on the type of cognitive processes required 
to complete an objective or answer a question. The cognitive levels are cumulative in structure; 
each level integrates and builds upon the cognitive activities of the levels below it, implying a type 
of sequence, or a hierarchy, to the levels of thinking. The six levels are: 

Knowledge - Involves the recall of specific facts and theories, methods and 
processes. This level emphasizes remembering learned material (Bloom, 1956). 

Common action terms for this level include: list, define, label, match and designate 
who, what and when. 

Comprehension - Represents the lowest level of understanding. The individual 
knows the information which is being communicated and can make use of the 
material without relating it to other information or seeing its fullest implications 
(Bloom, 1956). Common action terms for the comprehension level include: explain, 
paraphrase, summarize, rewrite, and give examples. 

A pplication - Focuses on having students apply what has been learned to different 
situations and learning tasks, requiring students to use information that they know 
and understand (Bloom, 1956). Common action terms for this level include: 
compute, demonstrate, use, predict, discover, and solve. 

Analysis - Involves breaking down the information or situations and separating them 
into their component parts, focusing on the relationships of these parts with each 
other and with the whole structure (Bloom, 1956). Common action terms for the 
analysis level include: differentiate, discriminate, relate, diagram, and distinguish. 

Synthesis - Focuses on the combination of learned elements and parts to form a new 
whole. This includes working with pieces and elements and arranging them so as to 
create a new form, pattern or structure of the information (Bloom, 1956). Common 
action terms for this level include: create, compose, produce, and develop. 

Evaluation - Involves making judgments about the value of material and methods 
for given purposes. Judgments are made based on standards or criteria, either 
established and provided for the student or those determined by the student (Bloom, 

1956). Common action terms for the evaluation level include: justify, compare, 
contrast, evaluate, and interpret. 

The first two levels, knowledge and comprehension are typically referred to as lower order 
thinking and the four highest levels (application, analysis, synthesis, and evaluation) represent 
higher order thinking as they involve more complex processing. Although Bloom’s taxonomy is 




recognized as not being perfect, it has been widely accepted and used by many educators and 
researchers. 



Purpose and Objectives of the Study 

The purpose of this study was to cognitively assess the academic challenges that were 
incorporated into selected courses offered within the College of Agricultural Sciences at the 
Pennsylvania State University. The following objectives guided the study: 

1 . To describe the various types of academic challenges and the frequency with which 
each type was used in College of Agricultural Sciences courses. 

2. To determine the cognitive level at which the students were challenged to think for 
each academic challenge. 

3. To examine the cognitive levels of the academic challenges and their value to the 
students’ final grade. 

This study was designed to provide educators with a system to collect benchmark 
information on the academic challenges they were using. It is hoped that this system will 
contribute to an increase in professors’ effectiveness in writing academic challenges which 
facilitate the development of their students’ higher order thinking ability. 

Procedure 

Eleven faculty members from nine departments/schools within the College of Agricultural 
Sciences (agricultural economics, agricultural education, agronomy, animal and veterinary 
science, dairy science, entomology, forestry, horticulture, and plant pathology), participated in 
this project. Faculty members were initially nominated for participation by their department 
chairperson who categorized potential participants as “good” teachers. Final participation in the 
project however, was voluntary. Each participant provided copies of all the academic challenges 
used in their course. The participating professors were consulted when necessary, to provide 
background content information and to clarify any questions raised during the analysis. 

The collected academic challenges for each course were examined and categorized by type 
of challenge. The resulting categories were: activities, problem sets, written reports (individual 
and group), presentations (individual and group), laboratory tests, quizzes, midterms, and finals. 

Each individual task, question, problem, or action within all of the academic challenges 
was analyzed to determine the cognitive challenge it provided to students. A number 
corresponding to one of the six levels of Bloom’s taxonomy (i.e., 1 = Knowledge, 2 = 
Comprehension, 3 = Application, 4 = Analysis, 5 = Synthesis, and 6 = Evaluation) was assigned. 
The analysis was based on the highest level that students would be cognitively challenged in order 
to answer the item. 
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In addition to the cognitive assessments, the value of the academic challenge to students’ 
course grade was calculated. This was based on the grade weighting information provided in each 
course’s syllabus which detailed the value of all the academic challenges provided to the students’ 
final grades. 

The collected data were compiled and entered into a spreadsheet which contained the 
complete listing of the academic challenges provided in the course and each academic challenges’ 
cognitive distribution. The cognitive distribution was based on the cognitive assessment of the 
academic challenge. It describes the portion of each cognitive level included in the academic 
challenge. For example, a problem set may have a cognitive distribution of: 0% knowledge, 7. 1% 
comprehension, 71.4% application, 7.1% analysis, 0% synthesis, and 14.4% evaluation. In other 
words, 7. 1% of this example was written to challenge students at the lower cognitive levels and 
92.9% challenged the students at the higher cognitive levels. From this spreadsheet, the courses’ 
mean cognitive distribution for the academic challenges provided was calculated. 

A second spreadsheet was used to calculate the grade-weighted cognitive distribution. 
While similar to the above spreadsheet, this spreadsheet included the value of each academic 
challenge to the students’ final grades. To obtain the grade weighted cognitive distribution, the 
initial cognitive distribution for each academic challenge was multiplied by its value to the 
students’ final grade. 

Several of the courses provided academic challenges which contained individual items 
with different point values (i.e., one question worth 10% of the academic challenge’s grade and 
another question worth 15%). These values were included in the analysis by re-calculating the 
cognitive distribution for that academic challenge such that it included the weighted value of each 
item. Then the weighted cognitive distribution for the course could be calculated as described 
above. 



Results 



Types of Academic Challenges Provided 

As can be seen in Table 1, there was a wide range of both the total number of academic 
challenges and the number of different types of academic challenges provided to the students. 
The total number of academic challenges ranged from a low of 6 to a high of 29 academic 
challenges. The mean number of academic challenges provided by the 1 1 professors was 13 .2. 
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Table 1 

Academic Challenges Provided By Course 



Course 



AC category 


A 


B 


C 


D 


E 


F 


G 


H 


i 


j 


K 


Activities 




1 




2 






1 


7 


6 


19 


n 

J 


Problem sets 






5 








14 




1 


1 




Written reports, ind 




4 


4 


10 


1 


1 


1 




4 


4 




Written reports, grp 


2 










7 




i 


2 


i 




Presentations, ind 












4 












Presentations, grp 


i 






1 




1 




1 








Quizzes 


3 








3 




2 






3 




Laboratory test 


1 






















Midterm exams 


1 


1 


2 


3 


2 


1 


2 






1 


2 


Final exams 


1 


1 


1 


1 


1 


1 


1 








1 



Total# of AC provided 9 7 12 17 7 15 21 9 13 29 6 

mean = 13.2 

# of categories used 64454663463 
mean = 4.6 



The number of different types of challenges provided also varied although not as much. 

On the low end, two professors utilized 3 different types of challenges. The greatest variety of 
academic challenges was provided by four professors who utilized 6 different types of challenges. 
The mean number of different types of academic challenges provided by the professors was 4.6. 

Cognitive Distribution 

The cognitive analysis by course provided information on where the cognitive emphasis 
was placed for each of the course’s academic challenges (Table 2). The mean cognitive 
distribution for all of the courses’ academic challenges was fairly even across the six cognitive 
levels with an emphasis on the higher cognitive levels. The examination of all the courses resulted 
in means of 29.6% written at the lower cognitive levels (with a range of 12.2% to 77.3%) and 
70.5% written at the higher cognitive levels (with a range of 22.7% to 87.8%). The mean course 
cognitive distribution was: 11.5% at the knowledge level, 18.1% comprehension, 19.8% 
application, 15.8% analysis, 17.7% synthesis and 17.2% evaluation. 
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Table 2 

Cognitive Analysis by Course 



Cognitive distribution Mean % of students' grade 



Course 


n 


K 


C 


Ap 


An 


S 


E 


K 


C 


Ap 


An 


S 


E 


A 


9 


3.0 


20.3 




40.9 


29.6 


6.1 




9.4 


20.1 




23.5 


28.9 


18.5 


B 


7 


5.7 


11.4 


14.3 


22.9 


10.4 


35.3 




4.0 


9.3 


10.0 


16.0 


15.5 


45.3 


C 


12 


8.3 


9.1 


47.4 


14.3 


13.1 


7.8 




16.6 


19.5 


30.7 


17.0 


13.2 


3.7 


D 


17 


10.1 


18.2 


10.0 


11.6 


28.8 


21.4 




16.5 


22.3 


8.3 


18.2 


17.3 


16.6 


E 


7 


50.9 


26.4 


6.5 


0.4 


13.7 


2.0 




42.5 


16.4 


7.8 


1.4 


28.1 


3.7 


F 


15 




35.3 


12.3 


3.8 


19.8 


28.7 






24.1 


18.9 


6.0 


21.8 


29.8 


G 


21 


5.4 


10.6 


58.6 


5.9 


2.5 


17.1 




11.6 


27.9 


30.3 


10.0 


2.7 


17.8 


H 


9 


7.6 


9.5 


11.1 


28.3 


16.4 


27.1 






6.1 




20.1 


47.4 


26.7 


I 


13 




12.2 


11.5 


16.9 


27.1 


32.2 






5.2 


9.1 


15.8 


36.8 


33.0 


J 


29 


20.4 


18.7 


20.5 


15.0 


18.0 


7.4 




21.3 


15.0 


12.2 


7.6 


16.0 


4.0 


K 


6 


14.6 


27.0 


25.8 


13.9 


15.0 


3.7 




21.1 


36.7 


17.7 


12.6 


9.0 


2.8 



K C Ap An S E 

Mean %: 11.5 18.1 19.8 15.8 17.7 17.2 

K C Ap An S E 
Total % of final grade: 13.0 18.4 13.2 13.5 21.5 18.4 



The overall grade-weighted cognitive distribution shifted the cognitive distribution to: 

13% knowledge, 18.4% comprehension, 13.2% application, 13.5% analysis, 21.5% synthesis and 
18.4% evaluation. The result was that on average, 3 1.4% of the students grade came from work 
at the lower cognitive levels and 66.6% came from work at the higher cognitive levels. 

Several courses rewarded the students more for the portion of academic challenges written 
at the higher cognitive levels. For example, the academic challenges provided in Course H 
emphasized the higher cognitive levels (a mean of 82.9% written at the higher cognitive levels) 
and the course’s grading structure was set such that over 93.9% of the students’ grades came 
from the work at the higher cognitive levels. Similar shifts were observed in Courses B, E, F, I 
and J. Conversely, in Courses A, C, D, G and K the opposite occurred, where the lower 
cognitive levels were worth more than their initial portion of the cognitive distribution. For 
example, 16% of the academic challenges in Course G was written at the lower cognitive levels, 
but that 16% was worth 39.5% of the students’ final grades. Much of the shift observed in 
Course G was due to the weight of the quizzes, midterms and the final exam which all emphasized 
the lower cognitive levels but were worth 80% of the students’ final grades. 

The analysis by type of academic challenge also revealed several differences in the 
cognitive distributions (Table 3). On average, the individual presentations, quizzes, midterm 
exams, and final exams tended to provide lower cognitive challenges for students. These 
academic challenges were written with means of 87.5%, 69.5%, 65.9% and 52.6%, respectively, 
at the lower cognitive levels. This emphasis on the lower cognitive levels of knowledge and 
comprehension is similar to the findings of Newcomb and Trefz (1987). However, within each of 
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these categories, there are examples of challenges that were written to emphasize the higher 
cognitive levels. 

The other academic challenge categories (activities, problem sets, individual and group 
written reports, group presentations, and laboratory tests) tended to be written to emphasize the 
higher cognitive levels. Problem sets placed the greatest emphasis on the higher cognitive levels 
(a mean of 97.2%). The activities had some emphasis on the lower cognitive levels (a mean of 
27.8%) but were still written to emphasize the higher cognitive levels (a mean of 70.2%). Based 
on the cognitive analysis, it appears that these non-exam types of academic challenges, by their 
nature, are more conducive to challenging students at the higher cognitive levels of application, 
analysis, synthesis and evaluation. 



Table 3 

Academic Challenge Cognitive Analysis 



Cognitive distribution Mean % of students' grade per AC 

Academic challenge n K C Ap An S E K - . Ap An S E % Grade 



Activities 


39 


12.4 


17.5 


24.7 


21.4 


7.8 


16.3 




0.3 


0.5 


0.9 


0.8 


0.5 


0.6 




3.5 


Problem sets 


21 


1.2 


1.6 


81.4 


2.3 




13.5 






0.1 


1.1 






0.1 


1.3 


Written reports, ind 


29 


0.5 


5.1 


9.8 


17.2 


44.0 


23.3 






0.1 


0.8 


1.3 


2.9 


1.9 


7.0 


Written reports, grp 


13 




11.5 


3.8 


7.3 


40.4 


36.9 




0.5 


0.3 


1.0 


3.1 


2.1 


7.0 


Presentations, ind 


4 




87.5 


6.3 


6.3 








2.9 


0.5 


0.5 






3.9 


Presentations, grp 


4 




12.5 


0.5 


52.1 


22.9 






0.6 


1.0 




10.0 


5.0 


16.6 


Quizzes 


11 


32.1 


37.4 


2.4 


24.1 


3.0 


0.9 


1.3 


1.8 


0.1 


0.5 


0.2 


0.1 


4.0 


Laboratory tests 


1 




12.5 




54.2 


12.5 


20.8 




1.9 




8.1 


1.9 


3.1 


15.0 


Midterm exams 


15 


33.1 


32.8 


8.5 


12.1 


4.0 


9.5 


4.8 


5.1 


2.2 


2.2 


1.3 


1.5 


17.2 


Final exams 


8 


22.9 


29.7 


6.5 


9.1 


15.5 


16.3 


5.7 


7.1 


2.4 


1.8 


3.0 


4.5 


24.5 



K C Ap An S E Total: 100.0 

Mean %: 10.7 17.7 22.3 15.8 16.7 16.4 

K C Ap An S E 
Total % offlnal grade: 12.1 20.6 9.3 16.2 22.9 18.9 



The examination of the types of academic challenges did lean toward the rewarding 
students more for completing academic challenges written at the higher cognitive levels and less 
for lower cognitive work, but not entirely. Academic challenges such as the individual and group 
written reports, the group presentations and the laboratory tests emphasized the higher cognitive 
levels and when combined were worth a total of 45.6% of students’ final grades (7 %, 7%, 16.6% 
and 15% respectively). Quizzes and individual presentations tended to emphasize the lower 
cognitive levels and were worth a total of 7.9% of students’ final grades (4% and 3.9% 
respectively). 

However several types of academic challenges were counter to the ideal. As was noted 
previously, the midterm exams and final exams tended to emphasize the lower cognitive levels but 
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they were worth 41.7% of students’ final grades (17.2% and 24.5% respectively). Activities and 
problem sets were worth a total of 4.9% (3.6% and 1.3% respectively) although they were 
written to emphasize the higher cognitive levels. This parallels Pickford and Newcomb’s study 
(1989) which concluded that activities tended to challenge students cognitively, but were 
rewarded the least. 



Discussion and Recommendations 

The focus of this study was to examine the academic challenges provided to students, how 
students were being cognitively challenged and how students were being rewarded, not to 
compare one course to another; this study was not a judgment of the choices made by professors. 
Instead, the assessment process is intended to provide benchmark information which may assist 
professors in examining the cognitive level of their academic challenges, how they are written, and 
how they reward students. It provides an opportunity for professors to question whether or not 
their academic challenges are accomplishing what they are intending. 

This study establishes a framework that educators can use to analyze their academic 
challenges and to assess whether or not they are challenging students at the cognitive levels to 
which they aspire. It is hoped that this framework will assist educators to improve their 
effectiveness at developing students’ ability to think at higher cognitive levels. 

The cognitive distributions varied by course and by type of academic challenge as they 
should, given the variety of courses examined. The observed variation provides examples of the 
range of possibilities for using academic challenges. Because of differences in the content and 
structure of the courses, comparisons between the courses should not be made except to provide 
examples of the different ways to structure academic challenges and the different types of 
academic challenges that can be provided. 

The observed variation between courses and within the academic challenge categories has 
generated more questions than answers. Recommendations for additional research include 
exploring questions such as: 

• Is there a specific combination of types of academic challenges that would be most 
effective for developing students’ thinking ability within a particular course? 

• What number of academic challenges should be provided in a given course for 
maximum effectiveness? 

• Given the recognized importance of developing students’ ability to think at the 
higher cognitive levels and given the hierarchical nature of Bloom’s taxonomy, is 
there a “best” distribution of the six cognitive levels for writing academic 
challenges? 

• Does rewarding students more for work completed at the higher cognitive levels 
encourage the development of their thinking ability? 

• How do student variables (such as attitude, motivation, challenge acceptance) 
affect the development of students’ thinking ability? 



• What interventions can be used to improve how professors are writing academic 
challenges to increase the cognitive challenge to students? 

• Can this cognitive analysis process be used as a tool for detailing professors’ 
teaching activities and therefore be used in promotion and tenure reviews? 
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